DOCOBEIT BSSOHE 



BD 206 528 



£A 014 074 



AOTBOR 
TITLE 

POB DATE 
NOTE 

BDBS PRICE 
DESCRIPTORS 



King, Richard Pohland, Paul A« 

K Cost^Effectiveness Analysis of Administrator 

Training Programs for Native Aiericans* Uonograph* ^ 

81 

39p^ 

HPbl/PC02 Plus Postage. 

^Adninistratof Education; ^Anerican Indians; ^Cost 
Effectiveness: ^Educational Adainistration; 
Evaluation Criteria; Higher Education; Program Costs; 
Program Design; Program Effectiveness; Program 
Evaluation 



ABSTRACT 

\ Researchers analyze the cost effectiveness of 
educational administrator training programs for Native Americans at 
four universities: Harvard, Pennsylvania State, Minnesota, and Nev 
Herico« The ps^ograms vary in scope, duration, clientele, admissions^ 
and five other characteristics* The programs* average costs per 
student are computed and- are related to « differences in program 
characteristics.. The researchers then compile effectiveness rankings 
for seven different outcome measures, including program completion 
rate, post^program jobs and salaries, shifts in job aspirations, and 
students* percepxions of program quality, chaiages in their 
decision-making responsibilities, and realiaaticn of preprogram 
expectations. Pro^gram rankings on each outdome are weighted and 
averaged, and a measure of program cost per unit of ef f^^ctiveness is 
computed. Dhile one unidentified school is found to have lover costs, 
the authors conclude that no school can be considered more cost 
effective because xhe use of> different outcomes or veights vould 
yield different figures for cost effectiveness. They discuss the 
problems of analyzing the cost effectiveness of social action 
programs, especially in relation to variation in program goals, 
structures, resource constraints, -and outcome measures* 
(Author/RW) 
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A COST-EFFECTIVENESS ANALYSIS OF 
ADMINISTRATOR TRAINING PROGRAMS 
FOR NATIVE AMERICANS 



As an evaluation methodology, cost«-effectlveness analysis Is 

j 

designed to assist in the determination of which among several feas- 
.ible courses of action will utilize resources most effectively. 
This methodology has been accepted by educational decision-makers 

as an aid in selecting policy alternatives on the basis of least 

1 

cost and greatest effectiveness. The study described in this ar- 

tiicle applies the methodology of cost-effectiveness analysis to four 
2 

graduate programs designed to prepare Native Americans to assume 
leadership roles in education. 

As stated by Levin, cost-effectiveness takes as its focus the 
determination of "that strategy or combination of strategies that 
maximizes the desired result for any particular resource or budget 
constraint.""^ Embedded within this statement are four variables 



^Terry G. Geske, "Some Observations on Cost-Effectiveness Analysis 
in Education," Journal of Education Finance 4 (Spring 1979), pp. 451-468. 

^Programs compared in this analysis were located at Harvard Univ- 
ersity, The University of Minnesota, The Pennsylvania State University, 
and The University of New Miexico.. The major data sources are the "Eval- 
uation Report of Indian Education Administrator Training Program at 
Universities of Harvard, Penn State and Minnesota" (Albuquerque: Indian 
Education Resources Center, Bureau of Indian Affairs, 1975), and compar- 
able sources from the University of New Mexico program. Programs are 
presented herein as A, B, C and D to focus attention on programmatic 
differences and the methodology of cost-effectiveness rather than on 
institutions. 

^enry M. Levin, "Cost-Effectiveness Analysis in Evaluation Research, 
Handbook of Evaluation Research , eds. M. Guttentag and E.L. Struening 
(Beverly Hills, Ca.: Sage Publications, 1975), p. 89. 



germane to cost-effectiveness analysis: goals C^the desired result"), 
alternatives ("strategies"), constraints ("resource or budget") and 
outcomes or effectiveness ("maximizes the desired result") • Each of 
these variables are addressed in the context of the four institutional 
pr6grams analyzed. 

GOALS OF ADMINISTRATIVE TRAINING PROGRAMS 
Each of the training programs adopted as a primary goal the 
preparation of personnel with necessary skills to assume leadership 
positions in the field of education. Programs emerged from a 

pressing need to prepare selected Indian persons with high 
potential leadership abilities and provide them specialized 
training in management, administrative and change agent skills 
which woxild be. utilized in educational systems that directly 
affect Indian people. 

While this goal statement is indicative of institutional responses 
to the national social consciousness prominent in the 1970 *s, 
institutions found modifications appropriate. Goals specific to the 
alternative programs are as follows. 

Program A ; to produce graduates in educational administration 
qualified to accept management and/or leadership positions in 
schools serving large numbers of Indian children. 

Program B ; to provide opportunities in obtaining change agent 
skills . 



^"Evaluation Report of Indian Education Administrator Training 
Program at Universities of Harvard, Penn State and Minnesota" 
(Albuquerque: Indian Education Resources Center, Bureau of Indian 
Affairs, 1975), p*ii. 



Program to prepare American Indians for administrative positions 
in higher education « 

Program D ; to train twenty [Native Americans] at the Masters 
^ level as administrators in the various types of schools serving 
[Native American] students. 

Focusing the discussion on "program goals" clearly limits this analysis. 

An exhaustive goal-based evaluation would have to include the multiple 

goals of Indian tribes, funding sources, university graduate schools, 

colleges and departments, varied educational agencies, and students 

themselves^ Constraints defined by each of these agencies and groups 

do, however, account for variations in goal statements and for variations 

- in programs, or in the language of cost-effectiveness, the generation 

of alternatives. 

ALTERNATIVE STRUCTURES FOR 
ADMINISTRATIVE TRAINING PROGRAMS 

e 

The interplay of a common goal and unique constraints generated 
"•^our institutional alternatives for the training of Native American 
school administrators. Data presented in Table 1 illustrate differences 
and similarities along common dimensions. 

Insert "Table 1 About Here 

Variable one, program model, identifies all four as being "nested", 
i.e., as programs within existing programs. ^ However, the degree of 

^Bill Burgess, William H. Holloway .and Jerry E. Hutchinson, 
"Educational 'Leadership Development: The Contribution of One 
University,^' paper presented at Annual Meeting of American Education 
Research Association, Toronto, Canada (1978). 



: TABLE 1 



INSTITUTIONAL ALTERNATIVES FOR TRAINING 
NATIVE AMERICAN ADMINISTRATORS 



Source of . Program 

Variation ' . • 







A 


B 


C ' 


D 




Program Model 


Nested 


Nested 


Nested 


Nested 


2. 


Scope 


M.A. 
Ed. S. 
Ph.D. 


M.Ed. 
Ph.D. 
Ed.D. 


K. Ed. 

CAS 

Ed.D. 


M.A. 


3. 


Residency 


Full- 
\tiMe 


Full- 
time 


Full- 
time 


Part-time 


4. 


Duration 

(M^A./M.Ed. 

Program) 


One 

academic 
year 


One 

academic 
year 


One 

academic 
year 


Two calendar 
years 


5, 


Instructional 

Delivery 

System 


Tradl- 
tlonal 


Tradl- 
tlonal 

V 

Multi- 
tribal 


Tradl- 

^ 4 MM A 1 

Clonal 


Innovative 
^i-D-l-D-i; 


6. 


Client 


, Multl- 
trlbal 


Multl- 
trlbal 


Single tribe 


7, 


Locus of 
Admission 

Decisions 


"Special 
committee" 


Program 
director 


Institu- 
tion 


Shared: trlb< 

' and 
Institution 


8. 


Source of 
Funding 


Federal 
government 


Federal 
government 


Federal 
government 


Private 
foundation 


9* 


Fiscal 
Control 


Institu- 
tion 


Institu- 
tion 


Institu- 
tion 


Tribe 
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"nestedness" varied considerable. For example, program C contained 
'•no training or courses designed specifically for the Indian students" 
and students, with advisor assistance, planned their own programs of 
study. In brief , students were "nainstreamed" , i.e., absorbed\ 
individually into the existing training program. Further, program C 
ftmctioned without a director and "primarily as a social organization 
for Indian students". In contrast, program D modified existing courses 
and devised new learning experiences, adopted a group rather than 
individual approach to programming and Instructional delivery, had a 
director, and participants gradually acquired a unique identity within 
the university community. Programs A and B fell between these two . 
extremes . 

While all programs were "nested", they varied with respect to 
scope, residency requirements, duration, and instructional delivery 
syijtem. Programs A and C were designed to offer the Masters, Certificate 

e 

of Advanced Study (or the Education Specialist), and the Doctorate. 
Program B offered the Masters and Doctorate, while program D was limited 
to the M.A. Given variation in scope, variation in residency require- 
ments followed. At the M.A. level, programs A,B and C required a 
full-time academic year residency with the expectation that the degree^ 
requirejnents could be fulfilled within that time frame. In contrast, 

the non-residency, part-time nature of program D required two calendar 

♦ 

years (6 semesters) for completion. The latter difference, stemming 
from -an early decision to minimize dislocation problems and enhance 
the probability of graduates assuming on-reservation work responsibilities, 
required the construction of a unique instructional delivery system, 
'^at envolved was, in program jargon, the "1-5-1-5-1" academic year 
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1 



6 



system; where one-week sessions of, full-time study at the university 
were followed by five intervening weeks of full-time employment in 
' professional work settings. In addition, students were rc-uired to 
attend two eight -week summer sessions on campus. Further refinements 
in. the instructional delivery system included regularly scheduled 
"on-site Saturday sessions" and "instructional contracts", i.e., 
assignments designed to provide the opportunity to apply administrative 
concepts studied during residency weeks in the immediacy of .the 
work setting. Clearly, this system varied substantially from the more 
traditional systems at the other three Institutions. 

Variation also existed relative co clients. While all students 
were Native An»ricans as determined by tribal census or blood quantum, 
programs A, ? '-id C recruited nationally and inter-tribally while 
program D was limited to a single tribe. Clearly, this had implications 
for recruitment practices. A greater percentage of participants in 
program A (74%) and program B (81%) were male than in program C (62%) 
and D (56%) . Students in program C were younger than those enrolled in 
other programs — AO percent of participants were 20-30 years of age 
while only 10 percent were over 40 years. In constrast, 13 percent 
of participants in program D were under 30 and 31 percent were over 
40 years of age. Similarly, students in program C had fewer years of 
professional experience (38% had 1-3 years and 2% had over 16 years 
of experience) as compared with students in program D- (2% had IrS 
years and 23% had more than 16 years of educational experience) . 
The non-residential nature of program D tended to attract older, more 
experienced career educators than did the others. 



ERIC 



8 



. Considerable variation j^as also present relative to admission 
processes. At program C there was no vai^iatlon from general imiverslty 
procedures « Program A handled admissions on a %lock basis" and 
upon the recommendation of a "special committee" comprised of five 
fac\alty members and the state Director of Indian Education, At 
program B, participant selection largely devolved upon the Program 
Director with input from faculty and students. More complex procedures 
used at program D involved review and screening by the tribal office 
of education, the BIA Area Office where appropriate, a program advisory 
Council, the faculty of the Department of Educational Administration, 
and the university Graduate School, In brief, admissions processes 
varied in complexity among the four training institutions, 

' The final sources of variation identified in Table 1 are related 
to fXscal matters. The funding source for programs A, B and C was 
the Federal government — OE and the BIA; for program D it was a 
private foundation — Carnegie Corporation of New York, More important, 
in the case of programs A, B and C, grants were made directly to the 
institutions; in the case of program p, funds flowed directly to the 
tribe with the university subcontracting for instructional services. 
Clearly, the latter condition altered the nature of the decision- 
making process, invoking shared decision-making to a degree not 
required by the other three Institutions, 

In concluding this discussion of "^UMfiations the authors wish to 
"foreshadow" several issues to be addressed later. First, the extent 
of variation among programs was, in all probability, greater than that 



9 



Indicated* Data limitations precluded speaking to such issues as scope 
and magnitude of academic demands, support services, faculty composition, 
facilities, and the like* » Given that, it is uncertain whether or not the 
most sensitive sources of variation have heen identified. Secondly, it 
may be argued that similarities among programs A, B and C are so pronoiinced 
as to constitute one rather than three alternatives. Hence the'' foreshadowed 
question of what within the framework of cost-effectiveness indeed constitutes 
"an alternative." 

COSTS OF TRAINING PROGRAMS 
As more educational programs have competed for fewer available re- 
sources during the past decade, the importance of examining both potential 
outcomes and potential costs of alternative strategies has led educational 
program evaluators to employ such methodologies as cost-benefit, cost-effec- 
tiveness, or cost-utility analysis.^ 

•> 

Each of these methodologies identify cost as a measure of inputs to 

the program being evaluated. *rhe importance of understanding the 

evaluator's conception of cost is pointed out by Haller whj suggests 

that costs represent '^benefits foregone" » 

A cost is a sacrifice of one benefit in order to attain another. 
Costs ocdur when a choice is made among several desired benefits. 
In short, costs are benefits — benefits given up by choosing to 
do one thing rather than another. ^ 



^An extensive disctission of the distinctions between these method- 
ologies is presented by Henry M. Levin, "Case Book on Cost Analysis 
in Educational Evaluation," Research on Evaluation Program Report No. 33 
(Portland, Oregon: Northwest Regional Educational Laboratory, 1979). 

^Emil J. Haller, "Cost Analysis for Educational Program Evaluation," 
Evaluation in Education , ed. W.J. Popham (Berkeley, Ca.: McCutchen 
Publishing Co., 1974), ?• 407. \ 
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Resources which are diverted to Implemeut? a chosen alternative are no « 
longer available for other competing programs. Opportunities are thus 
foregone when one program alternative Is chosen rather than another. 

Similarly, Levin defines costs as ^^th^t set of social sacrifices^ 
associated with any particular choice among social-policy alternatives" . 
Rather than focusing exclusively on the actual dollar costs of program 
alternatives, both Haller and ievin ^encourage evaluators to assess the 

losses to society, institutions, and individuals in the form of benefits 

* 

and opportunities foregone. 

In the present analysis of tralnii^ programs for Native Americans, 
it is clear that social sacrifices are inherent in decisions by the; 
government and other funding agencies to finance such programs. In 
addition. Institutional decisions to direct time, energy, and facilities 
to these programs have meant sacrifices of potential benefits which 
could }iave been derived from other training or research endeavors. 
Moreover, individual students and families have foregone potential 
income, while society has foregone tax revenue and work productivity 
from the students enrolled. 

It'^s difficult at jbest * to assess the value of all the benefits 
sacrificed and opportunities foregone by a decision to Implement a given 
alternative. OperatiX)nally , therefore, we have adopted Haller 's stance 
that when an evaltxator employs' a dollar ^amount it is assumed that 

^vln, "Cost-Effectiveness Analysis in Evaluation Research," p. 98 

I. 
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"that figure will serve as an adequate measure and description of the 
resources required, the ^programs foregone, and the val;ie of the foregone 
programs",^ Further, data limitations restrict our discussion only to 
actual .dollar costs of project administration, instruction, and student 
support • Clearly, there were other **costs." For example, differences 
in residency requirements rai'se the possibility that "earnings foregone" 
bay be one of the associated costs* Students at programs A, B and C 
were required to leave their employment for a-miaimm of one year of 
full-time study on campus, while students in program D were able to 
remain employed. It aj^wfars at first that" an analysis of earnings 
would reveal greater losses of salary to individttals and of taxes and 
productivity to society from the former programs than from the latter. 
Student and dependent stipends helped ease the individual sacrifices 
of full-time students in programs* A, B and C; replacements for those 
students who,/ left positions, diminished the impact of their loss,.a/ 
their institutions. On the other hand, although students in program Dx 
did not forego earnings while on campus*' for one-week sessions, other 
costs were borne by the institutions they worke^ox: ' loss of their 
productivity, cost of replacements, effects t)f disruption on pupils • 
by the loss of a teacher or principal every five weeks for two years, etc. 
Th^ task of identifying such costs to individuals ai^d institutions is 
beyond the scope of this study. 



0 

Haller, "Cost ^alysis for Educational Program Evaluation," p. 411. 
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Actiial per' student dollar costs of the four training programs 
are presented as Table 2 « When deflated to comparable dollab: values^ 
the average per student cost of a program of studies ranged from 
$8,188 for program B to $9,919 for program C, Marginal costs for 



Insert Table 2 About Here 



programs indicate that program C is 21,1 percent more costly than 
program 

Although average costs do not vary greatly among programs A, B 
and D, the proportion spent on project administration, instruction, 
stipends for students and dependents, and university overhead does 
fluctuate (see Tal)ie 3). The proportion spent on project administration 
and instruction was highest (53Z) for program A and lowest for 
program B (35Z) . The lower percentage spent on administration (14%) - 
for program C was offset by the higher proportion (29Z) spent on 
tuition; just the reverse occurred in program B where a higher 
proportion was spent on administration (27%) than on tuition (8%) • 
The higher average cost of program C (see Table 2) appears to 
be due both to the higher cost of tuition and high proportion of 
funds provided for student support, as indicated in Table 3. 



Insert Table 3 Aboot Here 



Students in program D did not pay tuition separately since all 
costs of instruction were accounted for through the project. Although 
the proportion of the totil^^^idget expended on project administration 
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lABLE 2 
PROGRAM COST PER STUDENT 



Program 



Year 



Total Cost 



Actual 



1972* 



Average Marginal Cost 

Program 

Cost (1972)° Amount % 



,1972-73 



$8,768 



$8,768 





1973-74 


8,102 


7,627 


$8,584 


$396 


4.8 




1974-75 ■ 


11,030 


9,357 








B 


1972-73 


6,076 


6,076 










1973-74 


9,699 


9,131 


8,188 








1974-75 


11,030 


9,357 








C 


1972-73 


9,664 


9,664 










1973-74 


12,272 


11,553 


9,919 


1,731 


21.1 




1974-75 


10,066 


8,539 








D 


1975-76 


5,709 


4,276 










1976-77 


6,190 


4,354 


8,221 


33 


0.4 




1977-78 


5,643 


3,703 









order to compare program costs, all were deflated to 1972 values 
using the Consumer Price Index, Bureau of Labor Statistics. 

( •■ ■ 

^Average cost of minimum time reqtiired for program completion: one 
year for programs A, B, and C, and two years for program D. 
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TABLE 3 

COSTS FOR ADMINISTRATION, INSTRUCTION, STUDENT 
SUPPORT, AND INDIRECT COSTS AS A 
PERCENT OF TOTAL COSTS 





A 


B 


C 


D 


Project Administration 










' and Instruction 


53% 


35% 


43% 


65% 


Project Administration 


30 


27 


14 




Tuition • • 




Q 
O 




l4oJ° 


Tribal Administration ' 


0 


0 


0 


19 


Total Student Support 


22% 


■ 54% 


41% 


31% 


Student Stipend 


17 


34 


33 


31 


Dependent Stipend 


5 


20 


8. 


0 


Indirect Costs 


7% 


7% 


7% 


4% 


Miscellaneous^ 


18Z 


4% 


9% 


0% 



^Breakdowns are not available since the project paid for all Instructional 
costs with tuition waived for students. 



^Represents costs not accounted for. 
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and instruction for program D was much higher (65%) than at the other 
institutions, these costs include those incurred by the tribal edu- 
cation division, thus representing the increased cost of "dual" control 

i ' ^ 
(i.e., 19% of total expenditures). Student and dependent support 

consumed a greater proportion of the budget for program B (54%) and 
program C (41%) than for program D (31%) and program A (22%) . 

In sum, it appears that programs B and C placed a higher emphasis 
on student support, while programs A and D expended a higher proportion 
of their budget on project administration and instruction. Ifore 
generally, in the language of cost-effectiveness, the total cost of program 
C represents the greatest sacrifice to society, i.e., it is 21.1% 
more costly to society to fund administrator training of the kind 
offered by program C lhan that offered by program B. 

EFFECTIVENESS OF TBAINING PROGRAMS 
As stated previously, cost-effectiveness analysis attempts to 
identify that alternative which maximizes the desired outcome. Typically, 
however, educational programs are characterized by multiple outcomes, 
hence increased complexity of assessment. Geske approaches that 
problem thus: 

Generally, a single criterion of effectiveness cannot 
adequately detect and estimate the possible effects of a 
program. In fact, even the measurement of progress toward 
attainment of a single objective often will require the 
use of multiple indicators. The assessment of program 
effectiveness is therefore typically based on a set of 
indicators or measurements.^" 



ERIC 



lOce^ke, "Some Observations on Cost-Effectiveness Analysis 
in Education," p. 459. 
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In Tables 4 through 9 which follow we oresent "a set of indicators" 
which bear on the effectiveness of the four Native American administrator 
training programs reviewed. As will become clear, the measures move 
on a continuum from hard/objective to soft/Impressionistic. The reader 
is cautioned regarding these measures of effectiveness due to incon- 
sistencies in the collection of program data and assumptions made by 
the authors in assessing program effectiveness. Indeed, as pointed 
out by Levin, differences in results of analysis may be due to "vagaries 
of evaluation procedures rather then differences in the actual program 
results. "^^ 

Perhaps the simplest effectiveness measure of academic programs 
is graduation data. Indeed, for institutions of higher education, 
completion of program/degree requirements may be the most salient 
measure of success. Table 4 presents data bearing on this measure. 



Insert Table 4 About l.?.re ^- ^ 



Several observations can be made about the program completion 
data displayed in Table 4. First, the variability in numbers of , 
participants cautions against direct comparability of completion 
rates. Secondly, only in the case of program D was a criterion 
established at the outset of the program — 20 M.A. graduates per 



^^evln, "Case Book on Cost Analysis in Educational Evaluation," p. 



TABLE 4 
PROGRAM/DEGREE COMPLETION 



Program 
ABC 



Number of Participants 48 64 47 61 

Program Completed 

Masters Degree 17 39 32 36 

Certificate of 

Advanced Study (CAS) — — 

Doctorate (EdD) 3 2 1 — 

Completion Rate 41.7% 64.1Z 70.2% 59.0% 



Effectiveness Rank 

Criterion I 1*3 4 2 

Criterion II * 2 . 2 2 4 
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funding cycle* Hence measured agaiast criterion, tba graduation rate 
might be reported more accurately as 36 of 40 or 90% for program b. 
Finally, program/degree completion rates for all institutions can be 
expected to rise given the number of current enrqllees. 

Consistent with the rationale of cost-effectiveness evaluation. 
Table 4 also presents a rank ordering of the effectiveness of the. four 
programs based upon the completion data displayed (4 « highest level 
of effectiveness). However, a criterion problem exists. If the percent 
of students admitted who completed the program is used (criterion I) , 
then program C was the most effective, followed by programs B, D and A. 
Conversely, if one uses program criterion as the index (criterion II), 
program D was the most effective (completion rate « 90%). Further, in 
the absence of any known program criterion, as was the case for 
programs A, B acd C, rank ordering convention in- the present case 
requires assigning "2" to each of those programs. 

While program/ degree completion may be most meaningful to training 
institutions, utilization of skills in actual lob placement is likely 
to be perceived as a more sensitive measure of effectiveness by Native 
American constituencies. Data on post-graduation employment are pre- 
sented in Table 5. 



Insert Table 5 About Here 



Again, several observations can be made. First, graduates of all 
programs are to be found in a wide range of agencies and offices. 
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TABLE 5 
POST-PROGRAM JOB PLACEMENT 



Program 

A B C D 

(N-23) (N-42) (N-40) (N-39) 



1. Federal Agency /Office^ .1 1 3-0 

2. State Agency /Off ice 0 110 

3. Local Education Agency^ 

Ceiitral Office 4 14 ,4 . 9 

-. School Site 3 5 0 16 

'^Classroom 0 3 0 2 

4. Higher Education 3^ 4 4 2 

5. Tribal/Inter-tribal -2 1 3 1 

6. Research Agency/Office 0 0 2 2 

7. Private Consultancy 2 0 2 1 

8. Program Adm. (Miscellaneous) .2 5 2 0 
'9. Student 0 2 2 0 
10. No Response 6 6 17 6 



Placement Rate 

X of Respondents 73.9% 81.0% 52.5% 84.6% 

% of Participants 35.0 53.0 44.7 54.1 



Effectiveness Rank 

• Criterion I 1 3.5 2 3.5 

Criterion II .2.5 2.5 1 4 



aExcludes BIA field level offices. 

^Includes boarding schools, contract schools, etc. 
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Second, consistent with espoused and implicit program goals, graduates 
of program B and C tended to gravitate toward high-level policy 
making roles, while graduates of programs A and D tended toward 
field-based roles. This is most noticeable in the case of pre .ram 
D where the "target role" was the" school principalship . 

The relative effectiveness of the four programs as measured 
by job placement is indicated in Table 5. Ranks are based upon 
the percent of participants who indicated they had been placed in 
administrative positions (criterion I) , and upon placements in 
positions directly related to program goals (criterion II) . Overall 
placement rates of participants were higher for programs A and C 
than for programs B and D. On the other hand, if placement rates 
are more narrowly construed as goal-congruent, program D (central 
office and school site administration) outranks program C (higher 
education). As indicated, placement rates for programs A and B 
according to program goals fall between. 

A related job placement indicator of program outcomes can be 
obtained by comparing the pre-program positions held by participants 
with their post-program preferences. Insofar, as "preferences" 
may or may not be realized in the market place, it may be more 
accurate to describe any shifts which might occur as changes in 
aspiration level. However, raising aspirations has long been touted 
as an attribute of American education, particularly higher education, 
and therefore might properly be used in the present context. Data 
bearing on this phenomenon are presented in Table 6 and indicate 



Insert Table 6 About Here 
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TABLE 6 

POSITIONS HELD PRIOR TO PROGRAM ENTRY 
AND POST-PROGRAM PREFERENCES 



Program 
B C 



Pre Post Pre Post Pre Post Pre Post 



Local -Education Agency 

School Administrator 
Education Specialist 
Teacher 

Higher Education 

College Instructor 
Student 

Trlbe/Inter-Trlbal 

Unclassified 

Program/Project Adm. 
Educ. Researcher 

Other/Not Sure 



40 
39 



2Z. 



13 

0 



4Z ft3)Z 0% (29)% 13% 37% 5% (85)% 

4 1) 10 13 10 25 5 18 

17 — ^ 46 — 30 — 40 3 



14 
?4) 



13 

10 

25 11 



15 



@ 
65 



44 

25 

31 30 



30 
20 



13 
10 



Effectiveness Rank 



Note: Columns may not total 100% since students were permitted to 
select more than one response; circled percentages indicate > 
shift consistent with program goals. 



*A dash (-) indicates that the option was not included in questionnaires. 
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quite clearly that niajor changes in participants' aspiration levels 
occurred across Institutions, The direction of change was decidedly 
upward, from teaching to administrative roles and from local education 
agencies and offices to more cosmopolitan settings, e.g., tribal offices 
and institutions of higher education. More broadly, the general 
programmatic thrust toward leadership roles appears to have been attained. 

Effectiveness rankings based on shifts in aspiration as related 
to explicit program goals are also presented in Table 6. Program D 
which was designed to produce local school administrators is ranked 
highest (4) based on the 80Z shift in preference to school adminis- 
trative roles. Program A which sought to place graduates in high level 
field administration positions is ranked second (3) on the basis of 
the combined percentage (60%) of those roles circled; program B is 
assigned a rank of 2; and program C which had a higher education thrust 
is ranked last (1). 

As a second measure of aspiration, data on anticipated future 
earnings were obtained by comparing pre-program earnings of participants 
with salary levels expected upon completion. Data displayed in 
Table 7 indicate that the median salary earned by students prior to 
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their admission in programs A, B and C was in the $11,000 to $13,000 
range, while students entering program D earned slightly higher 
(median in the $14,000-$16,000 range). The higher incpmes earned by 
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TABLE 7 

SALARY LEVELS EARNED PRIOR TO AND 
EXPECTED AFTER PROGRAM COMPLETION 



B 



Program 



Salary Range 


Pre 


Post 


Pre 


Post 


Pre 


Post 


Pre 


Post 


Over $20,000 


9Z 


39% 


5% 


^33% 


5Z 


-28% 


10% ^^56% 


$17,000 - 19,000^ 


0 


/ 

/ ■;^13 


0 / 


/ 


5 


/ J15 




^ 31 


















$14,000 - 16,000 


9 V 
// 


^ 35 




31 


25' 


/ 25 


36^ 


10 ' 


$11,000 - 13,000 




4 


43^-^ 


12 


35^ 


18 


26 


0 


Less than $10,000 


26 


0 


33 


0 


25 


0 


5 


0 


Unemployed 


4 


0 


2 


2 


5 


2 


0 


^0 


No response 


4 


9 


0 


5 


0 


2 


0 


3 



Effectiveness Rank 

Criterion I 
Criterion II 



2.5 
3.5 



2.5 
3.5 



2.5 
1.5 



2.5 
1.5 



Note : Arrows indicate shift in median (— ^ ) , and shift in interquartile salary le\^el ) . 

^Salary ranges are not inclusive; it is assumed that ranges were interpreted as $17,000 - $19,999 etc. 



24 



25 



^ • 23 



students in program D are due partly to an inflationary factor during 
the four year period between surveys and partly, to the higher ages and * 
levels of prior experience of participants discussed earlier. Regard- 
less .of median pre-program earnings, median post-program salary aspi- 
rations rose two salary levels across all four institutions as Indicated 
In Table 7. Hence In the effectiveness rankings based on the median 
(criterion\I) , each program was assigned the mean value of"^ four point 
scale' (2 J5). However, if the interquartile range rather than the median 
Is used (criterion II) , then some differentiation in effectiveness is 
noticeable. 

Program outcomes can also be asc^ttiined from a body of perception 
data. While such data are admittedly "soft^^ cost-effectiveness analysis 
permits their use. Two bodies of perceptior> data were generated: (1) ' 
those relatiye to.profesM.onal role enactment; , and (2)* those, relative 
to the quality of academit program exp^ienced. With respect to the 
first, students in all four programs were asked to compare. the level of 
responsibility and decision making authority they now enjoyed with that 
prior to program entry. Data on perceived changes are provided in , 
Table 8. " ^ 
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Consistent with data presented in Tables 6 and ^-7, program participants 

*^ . 

- h ^ 

across institutions perceived themselves as having ncreased responsibility 
and decision making authority commensurateJwith leadership position and 



TABLE 8 

PERCEPTIONS OF PRE- AND POST-PROGRAM RESPONSIBILITY 
AND DECISION MAKING AUTHORITY 



Program 



B 



D ^ 



Responsibility and 
Authority: 

About the same 
Increased 
Decreased . 
^ No response 



4Z 
65 

9 
22 



72 
71 

2 
19 



5Z 

58 
2 
35 



21Z 
72 

5 

3 



Effectiveness Rank 
Criterion I 
Criterion II 



2 

2.5 



3.5 

4 



1 

2.5 



3.5 
1 



salary level* Effectiveness rankings based solely upon this factor 
are presented as criterion I* (The trivial percentage difference be- 
"^tween programs B and D resulted in a 3 • 5- rank for each,) Utilization 
of criterion II (percent of "Increased" less percent of -"decreased" 
plus "about the same") results in a different rank ordering. Criterion 
II, however, is suspect given the large variation in-the '?no response" 

The final sets of effectiveness data to be reported address partici- 
pants* perceptions of program quality. In brief, respondents were 
akked to make judgements about their academic experience both in terms 
of quality and the degree to which their pre-program expectations were 

met. These data are displayed in Table 9. 

e'^ 
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As is Indicated in Table 9, 90-95Z of all students at all 
four Institutions rated their ^programs as good or excellent, with a 
slightly higher proportion of students -from programs B and D ascribing 
"excellent" ratings than those of programs A and C. This is reflected 
in the effectiveness rankings. (Effectiveness computed as the per- 
centage of respondents indicating "excellent" less those reporting 
"inadequate" or "fair",) Sindlarly, the majority of students in pro- 
gram C (65Z) , program B (64Z) and program D (80%) responded that their, 
expectations were "definitely" or "very definitely" fulfilled; slightly 
fewer (47%) of the respondents in program A chose those options. These 
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TABLE 9 

PERCEPTIONS OF QUALITY OF PROGRAMS AND 
REALIZATION OF EXPECTATIONS 



Program ^ 



ABC D 



Quality of Programs: / 

Inadequate/ - • 4t ' OX 3Z OX 

Fiir • " 6 5 5 5 

Good' " 43 40 55 33 

Excellent 47 55 - 38 62 



Effectiveness Rank '2 3 1 4 



Expectations Realized: 



Not at all OZ 5Z OZ 3X 

Minimally 9 10 5 0 

Generally 30 -7 25 18 

Definitely 30 40 35' 49 

Very definitely 17 24/ 30 31 

No response 13 14 5 0 



Effectiveness Rank 1 



2 



3 



4 



differences are reflected in the rankings assigned • (Rank of effective 
ness based upon percentage of respondents indicating "definitely" and 
"very definitely" less those reporting "not at all" and "minimally,") 



Comparisons of alternatives under a cost-effectiveness framework 
assist a decision maker by pointing out programs which derive a given 
level of effectiveness for^tbe. least, cost pr which yield the greatest 
..^ level of effectiveness at a given cost level. The four training programs 
are compared in Table 10 along both cost and effectiveness dimensions, 

Geske suggests that decision makers should prioritize program goals 
and should use a weighting scheme "to establish the relative Importance 
of the different program objectives,"^^ jh^g weights (ranging from 1 
to 4) were assigned to each of the m^^^es of effectiveness presented 
in Tables 4 to 9, Weighted ranks for each of the four programs were 
computed from which average weighted xanks were obtained. Average 
weighted units of effectiveness ranged from 31,5 and 32,5 for programs 
A and C, respectively, to a high of 51,5 for program D, 

Average program costs per participant presented in Table 2 range 
from $8188 for program B to a high of $9919 for program C, A measure 
of program cost per average unit of effectiveness, indicated for each 
ptpgram in Table 10, ^ranges from $159,63 for program D to $314,89 for 
program C, 



COSTS AND EFFECTIVENESS OF ALTERNATIVES 
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TABLE 10 

COMPARISON OF PROGRAM EFFECTIVENESS AND COST 









A 


B 


C 






D 








I 


II 


I 


II 


I 


II . 


I 


II 


Variable Table 


Weight 


Rank WR* 


Rank WR 


Rank WR 


Rank 


WR 


Rank 


WR 


Rank 


WR 


Rank 


WR 


Rank WR 


. Program Completion 


4 


4 


1 4 


2 


8 . 


3 12 


2 


8 


4 


16 


2 


8 


2 


8 


4 16 


Job Placement 


5 


4 


1 4 


2.5 


10 


3.5 14 


2.5 10 


2 


8 


1 


4 


3.5 


14 


4 16 


_.Job Preference 


6 


2. 


3 6 






2 4 






1 


2 






4 


8 




"^Salaty^^Preferencc 


7 


2 


2.5 .5 


3.5 


.7 


2.5 5 


3,5 


7 


,2.5 


5 


1.5 


3 


2.5 


5 


1.5 3 


Responsibility /Authority 


8 


2 


2 4 


2.5 


5 


3.5 7 


4 


8 


1 


2 


2.5 


5 


3.5 


7 


1 " 2 


Program Quality 


9 


1 


2 2 






3 3 






1 


1 






4 


4 




E3cpectations Realized 


9 


1 


1 1 






2 2 






3 


3 






4 


4 




Total Weighted 
Rankb 


High 
Low 
Average 


39 
26 
32. 


* 

5 




50 
39 
44.5 


40 
23 
31.5 


60 
43 
51.5 


Program Cost per Student^ 


$8584.00 


$8188.00 


$9919.00 


$8221.00 


Cost per Average Unit of Effectiveness 


$264.12 


$183.98 


$314.89 


$159.63 


Marginal Cost per Unit of Effectiveness 


$104.49 


$24.35 


$155. 2fr 





\ 



aweighted rank. * 

l»Sum of highest or lowest weighted rank for^each variable using either criterion I or II, and average of high and low ranks. 
Cc.f. Table 2. 
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If a minimum level of effectiveness had been specified, say 40 
for Illustrative purposes, then programs B and D would be preferable. 
In addition, program B wotald be thfe least costly alternative for 
achieving that specified level of ef f ectlveness « On the other hand. 
If a budgetary constraint dictated that not more than $8750 per 
student was available for program^ expenses , only A, B and D would 
be viable alternatives. The greatest effectiveness given that cost 
constraint would be realized under the format of alternative pro- 
gram D. Moreover, the lowest..cost per average unit of effectiveness 
($159.63) is realized under program D. 

Despite a conclusion that program D appears to be the most cost- 
effective alternative, it shotild be pointed out that other analysts 
tiilng different measures of effectiveness, varying assumptions In 
comparing those measures Included herein, or applying other sets of 
weights for program objectives, might arrive at very different con- 
clusions. Such considerations .were kept in mind by the authors as 
programs were ranked using two assumptions to Interpret several of 
the Indicators of effectiveness and as high and low weighted ranks 
\ were averaged. 

CONCLUSION 

Given the data presented and our concluding caveats, none of 
the four administrator training programs reviewed can be identified 
categorically as the most (or least) cost-effective. Consequently, 
the findings of the analysis may be only marginally relevant to a 
decision maker. This being the case, the crucial question becomes; 



33 



What are the mitiimim conditions which must be realized if cost- 
effectiveness analysis is to be of utility to decision makers? We 
again turn to the four dimensions of cost-effectiveness — goals, 
alternatives, constraints and outcomes — to structure the discussion. 
Objective Comparability of Goals 

Administrative training programs for Native Americans are 
illustrative of the emergent social consciousness of the 1960 's and 
70 's. Viewed in that sense, such endeavors were social action pro- 
grams predicated upon a reallocation of values. ^3 Predictably such 
reallocation was debated and ultimately legitimated at the highest 
levels of society — the Federal government. Hence goal statements 
referred to earlier were essentially political expressions consenually 
arrived at, manifestly intended to. appeal to wide and diverse constit- 
uencies, amenable to multiple interpretations, and difficult to evaluate 
uniformly. Even when "programmatically" reduced, the "political" 
content remained high and the outcome specificity low. Clearly, 
classical "goals-based"-^ evaluation of a ^^8^ program under such' 
conditions is difficult; the comparative evaluation of multiple pro- 
grams even more so. 

Levin states that only programs with similar or identical goals can 
be compared under a cost-effectiveness framework. "'•^ Indeed, the analysis 
presented in the present study suggests that it may be difficult to compare 
educational programs which appear to be directed at a particular goal, 
yet in reality have a multiplicity of goals defined by the political 
contexts of diverse institutions, funding agents, and clients. 



^^vid Easton, A Framework for Political Analysis (Englewood Cliffs, 

N.J,: Prentice-Hall, 1965). 
l^Levln, "Case Book on Cost Analysis In Educational Evaluation," p. 16. 



The Presence of "Real" Altematlyea 

.Consideration was given to conceptually viewing programs A, B and 
C as one rather than three discrete alternatives for preparing Native 
American school administrators. Indeed, they arc funaamentally identical 
on eight of the ten dimensions identified in Table 1, However, problems 
of data aggregation and variations among the three programs "suggested 
otherwise. Further, all four programs .were similar with respect to 
program model, i.e., all were "nested" and all were externally funded. 
Consequently, since decision theory at large and co8t-effectiveaet,3 
analysis in fine are dependent upon alternatives, we raise the question 
of what constitutes an alternative with some seriousness. 

The present research effort suggests the following as potentially 
significant sources of variation in alternatives for higher education 
training programs. First, we propose variation in model. Burgess, 
et al., identify five: natural systems ("do it yourself"), scholarship, 
legislated morality, separate but equal, and nested. These models 
currently exist; more could be generated. Clearly, logistical problems 
would be severe in studies based on differences in models, but they 
are not insuperable. Second, we suggest a full-time/part-time residency 
stipulation. Our study .indicates that this is crucial relative to such 
issues as student dislocation, time required for completion, and cost 
(particularly in respect to student stipends and foregone earnings). Third, 
we Suggest locus of control. Interorganizational decision making is 
manifestly more complex and costly than institutional decision making 

^Burgess, et al., "Educational Leadership Development: The Contribution 
of One University." 
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whl:h la turn Is more complex and costly than individual decision making. 
Some data from the present study are available which speak to ctila issue 
but the Impact on effectiveness is unclear. 

Omitted from the above list are such variables as type of traiiing 
institution (private vs. public university), source of funding, scope 
of recruitment, scope of program, and the like (cf. Table 1). Variation 
along these and other dimensions may be signif iciant , but our Immediate 
impression is that they contribute marginally to variation in effective- 
ness. They may, however, substantively vary with respect to costs. 

All of the above raises a significant policy issue. Assuming that 
either costs or prescribed levels of ef f ecclveness can be held constant , 
as the methodology of cost-effectiveness requires,!^ the potential for 
a ^'planned variation*'^^ is clearly present. To engage in such planned 
variation, however, would require that a funding source (e.g., 
the Title IV desk of the Office of Indian Education) deliberately adopt 
as operational policy a cost-effectiveness evaluation stance » ftmd only 
those programs which conformed to the **planned variations** established, 
and involve evaluators in the initial planning stages rather than in a 
post hoc fashion in order to insure data comparability and adequacy. 
Clearly, such a policy stance would be difficult for any funding agency 
to adopt. Yet, it would seem reasonable to assume that neither would 
it be totally irrelevant. 



Geske, **Some Observations on Cost-Effectiveness Analysis In Education." 

^^Allce Rlvlin and Michael Tlnq>ane, eds. Planned Variation lA. Education : 
Should We Give Up or Try Harder? (Washington, D.C.: The Brookings 
Institution, 1975). 
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Resource Constraints 

Of the four variables identified as being particularly germane to 
cost-effectiveness analysis, that of resource constraints is potentially 
the most troublesome As identified earlier in this study, constraints 
were posed by a variety of sources Including policies of the universities, 
\ colleges and departments involved, families and employers of students, 
funding agents, and, in the case of program n, a particular tribe. 
While it may be possible to assiome a singular constraint or universalist 
set of constraints among programs in simulation studies, our experience 
. is that it presumes an unrealistic view of the world. 

In brief, we. question the adequacy of cost-effectiveness formulations 
which posit that "meaningful comparisons can be made between the cost of 
different alternatives for achieving a prescribed effectiveness level, 
or between the effectiveness of different alternatives for a fixed budget 
level. "^^ Such formulations suggest a simple two variable relationship — 
budget level and prescribed level of effectiveness . This analysis of 
educational training programs suggests that contextual constraints 
must be taken into account as a third critical element. Hence a more 
appropriate approach to cost -effectiveness might be: "Given condition A 
(constraints) what might be an acceptable effectiveness level given a 
fixed budget?" or, "Given a prescribed effectiveness level, what might be 
an acceptable cost under condition B?" Such an approach seems more 

^^Geske, "Some Observations on Cost-Effectiveness Analysis ,in Education," 
p. 453. 

^^Elliot Mlshler, •'Meaning in Context: Is There Any Other Kind?" 
Harvard Educational Review 49 (1979), pp. 1-19. 
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productive than engaging In the "context-stripping'' activity which 
has dominated educational research and evaluation. 

An additional problem with resource constraints involves the 
identification of axl costs associated with alternatives. As in this 
study, analysts often rely upon program expenditures to represent 
dollar costs as well as social- sacrifices incurred under alternative 
strategies. Dangers inherent in such an assumption are discussed by 
Levin: 

... a group of decision makers may rationally take into 
account only its own costs when making a choice among alternatives, 
but this does not mean that costs to other constituencies should 
be ignored In the overall evaluation. To the contrary, all 
costs should be reviewed for purposes of uncovering the true 
social sacrifice of resources associated with a given program 
and level of effectiveness.^® 

To this end. Levin suggests the use of a model for cost measurement in 
which the costs of various "ingredients" for each alternative are 
specified, and the agency or individual who bears the cost (or 
sacrifice) is indicated. 

A strict accounting of all costs (including sacrifices) incurred 
*by funding agencies, by sponsoring institutions, by external agencies 
and by students themselves is imperative in determining and comparing 
the true costs of alternative train^g programs. The problems with 
developing accurate cost information for each of the "ingredients," 
including both direct expenditures and costs shifted to other agencies 
or individuals, are obviously very complex. Nevertheless, if all costs 
had been identified for each program, and if costs had been determined 
using similar guidelines, perhaps greater 'differentiation among costs 



^^Levln, "Cost-Effectiveness Analysis in Evaluation Research," p. 99. 



of all alternative delivery systems would have been possible. 
Measures of Effectiveness 

Similar to the determination of resource constraints, difficulties 
are encotjntered in obtaining accurate measures of outcomes of educational 
programs. This is particularly true with studies such as the present 
one in which secondary data from independent appraisals of programs 
were eiiq)loyed in the analysis. The difficulties of determining program 
effectiveness given crude data, differing assumptions in gatiiering and 
analyzing data, and subjective weighting and interpretations of those 
effectiveness measures were realized in this analysis. 

The importance, yet difficulty, of identifying educational goals 
as well as attainment of those goals has been discussed. This present 
experience with cost-effectiveness analysis leads to questioning its 
utility in the evaluation of social action programs with their multiple 
goals, imcertainty of meaning and measurement of "effectiveness^',* under- 
developed technology for the assessment of cost, and contextual dependencies. 
It appears that extant evaluatio'n paradigms are unequal to the task 
of evaluating social action programs. Developing conceptually sound 
and operationally feasible social action program evaluation models should • 
be a major agenda item for the 1980 's. 



3S 



